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ABSTRACT 

Background: Emergency department (ED) crowding and prolonged waiting times have been associated 
with adverse consequences towards quality and patient safety. 

Objective: This study investigates whether the number of patients simultaneously present at the ED 
might bean indicator of unsafe waiting and at what threshold hospital-wide measures to improve patient 
outflow could be justified. 

Methods: Data were retrospectively collected during a 1-year period; all ED patients aged > 16 years, and 
triaged as ESI-1 or ESI-2 were eligible for inclusion. The number of patients simultaneously present was 
used as occupancy rate. Waiting time was considered unsafe if it was longer than lOmin for ESI-1 
patients, or longer than 30 min for ESI-2 patients. Differences in waiting time and occupancy between 
patients with safe and unsafe waiting times were analysed using the Mann-Whitney U test. The ability 
of the occupancy rate to discriminate unsafe waiting times was analysed using a receiver operating char- 



Results: The overall median waiting time was 5 min (IQR = 4-8) for ESI-1, and 12 min (IQR = 6-24) for 
ESI-2 patients. Unsafe waiting times occurred in 16.0% of ESI-1 patients (median waiting time = 17 min, 
IQR = 13-23), and in 18.9% of ESI-2 patients (median waiting time = 48 min, IQR = 37-68). The occupancy 
rate was a weak indicator for unsafe waiting times in ESI-1 patients (AUC = 0.625, 95%CI 0.537-0.713) but 
a fair indicator for unsafe waiting times in ESI-2 patients (AUC = 0.740, 95%CI 0.727-0.753) for which the 
threshold to predict unsafe waiting times with 90% sensitivity was 51 patients. 

Conclusion: The number of patients simultaneously present is a moderate indicator of unsafe waiting 
times. Future initiatives to improve safe waiting times should not focus solely on occupancy, and expand 
their focus towards other factors affecting waiting time. 

© 2014 Elsevier Ltd. All rights reserved. 



Introduction 

Over the last years, there has been sufficient evidence of the 
increasing problem of emergency department (ED) crowding 
worldwide. Numerous studies have demonstrated the adverse con- 
sequences associated with crowding, and a growing number of 
government and professional reports express the need to solve 
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the "ED crowding problem" (McClelland et al., 2011; Pines et al„ 
2011). It is generally recognised that factors associated with 
patient flow should be held responsible as the root cause of the 
problem. More specific, the difficulty in transferring ED patients 
to a hospital bed are seen as the most important factors (Pines 
et al., 2011). 

As long as hospitals, and by extension the entire healthcare sys- 
tem, are unable to provide suitable solutions, emergency depart- 
ments (EDs) will be faced with the consequences of crowding. 
These consequences include prolonged waiting times, a decreased 
quality of care, and an increased risk of adverse events (Hoot and 
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Aronsky, 2008). In an attempt to cope with the associated risks of 
prolonged waiting times, EDs have implemented triage systems. 
The main objective of a triage system is to rapidly identify the most 
vulnerable patients, those who cannot wait to be seen, and provide 
these patients priority access to the diagnostic-therapeutic process 
(Gilboy et al., 201 1 ). Therefore triage systems suggest target times 
for physician evaluation in high urgency situations. Although there 
are differences between the systems, they all agree that high ur- 
gency patients should have quick, if not immediate, access to 
appropriate care and treatment. 

Currently EDs are increasingly unable to guarantee safe waiting 
times for high urgency patients (Triggle, 2013). In an attempt to re- 
duce the potential adverse consequences associated with these un- 
safe waiting times, a taskforce was set up in our hospital. This team 
consisted of clinical staff, management executives and (external) 
patient safety experts. The objective of this team was to analyse 
the current situation and identify potential, hospital-wide, mea- 
sures to improve the outflow of ED patients. The launch of hospi- 
tal-wide measures requires a certain indicator of when action 
should be taken. Prior to this study, the criterion to indicate ED 
crowding was set at 65 patients simultaneously present in the ED. 
It was, however, not clear on what basis this value was established. 

The objective of this study was twofold. On the one hand this 
paper investigates whether the number of patients simultaneously 
present at the ED might be an indicator of unsafe waiting times. As 
a second objective, it considers at what threshold hospital-wide 
measures to improve patient outflow could be justified. 

Methods 

Design and setting 

This retrospective observational study was part of a larger qual- 
ity improvement program at the ED of a tertiary referral academic 
teaching hospital in Belgium. The ED has an annual census of 
approximately 55,000 patients with an average hospital admission 
rate of 36%. The latter is in accordance with the average national 
rate. Overall, the median length of ED stay is 217 min. The ED con- 
sists of an admission and treatment area with 16 cubicles, includ- 
ing 3 resuscitation cubicles and 4 separate cubicles for minor 
trauma patients. There is a dedicated space with 4 cubicles for pae- 
diatric patients, which is staffed with a doctor and a nurse trained 
in paediatrics. In addition, the ED contains a 25-bed observation 
ward, of which 7 beds are equipped for intensive care and serve 
as a waiting zone in times of intensive care unit (ICU) bed shortage. 
Since January 2009, the 5-level Emergency Severity Index (ESI) is 
used to determine the treatment urgency and priority of patients 
visiting the ED (Wuerz et al., 2000). The ESI seeks to accomplish 
two main goals: patient sorting and patient streaming (Gilboy 
et al., 2011). As such, the ESI aims to accomplish these goals by 
(1) determining which patient should be seen first and (2), what 
resources are required to determine the patients' disposition. The 
most urgent level, ESI-1, is assigned in case immediate life-saving 
interventions are required, which are clearly defined in the ESI 
manual (Gilboy et al., 2011). Patients presenting with high-risk 
conditions or severe pain/distress are assigned to ESI-2. Although 
examples are provided, the interpretation of the criteria is based 
on triage nurses knowledge and experience. The following three 
levels of the ESI are assigned based on the estimated number of re- 
sources that a patient needs to reach a dispositional decision. 

Recruitment 

All ED patients, aged > 16 years and triaged as ESI level-1 or le- 
vel-2 during a 1-year study period from January 2012 to December 



2012, were eligible for inclusion in the study. Patients presenting 
with mental health problems were excluded, because their care 
was provided in a separate area with dedicated staff. 

Variables and definitions 

The number of patients simultaneously present at the ED was 
used as occupancy rate. The number of patients present at the 
ED is recorded every 10 min by the hospital's computer system. 
For each included patient, the occupancy rate was calculated as 
the mean number of patients present at the ED during the first 
hour of attendance, starting from the time of registration. 

Waiting time was defined as the time (minutes) between regis- 
tration of the patient in the ED information system and placement 
in a treatment cubicle. The later is registered by a care provider in 
the information system. 

To define unsafe waiting times maximum waiting times for ESI- 
level 1 and ESI-level 2 patients were set. While the ESI does not 
indicate specific time intervals to physician evaluation, it suggests 
the following for ESI-level 1 patients "ESI level-1 patients are seen 
immediately because timeliness of interventions can affect morbidity 
and mortality" (Gilboy et al., 2011) and for ESI-level 2 patients 
"All level-2 patients are still potentially very ill and require rapid ini- 
tiation of care and evaluation. The triage nurse has determined that it 
is unsafe for these patients to wait. Patients currently may be stable, 
but may have a condition that can easily deteriorate; initiation of 
diagnostic treatment may be time sensitive (stable chest pain requires 
an ECC within 10 minutes of arrival ); or the patient may have a poten- 
tial major life or organ threat. ESI level-2 patients are still considered 
to be very high risk" (Gilboy et al., 2011). Based on a pragmatic 
interpretation of these recommendations, unsafe waiting times 
were defined as follows: for ESI-level 1 patients a maximum of 
10 min waiting time and for ESI-level 2 patients a maximum wait- 
ing time of 30 min. The other ESI categories were not taken into 
consideration, as from a clinical standpoint these patients are con- 
sidered stable and can wait several hours before being seen by a 
physician or care provider. 

Data collection 

Data was collected from the hospital information system (clin- 
ical work station - University Hospitals Leuven). This system al- 
lows the extraction of detailed process times on patient level. 
First, all eligible patients were selected and variables regarding tri- 
age, waiting time and time of registration were obtained. A second 
dataset, containing the number of patients simultaneously present, 
was used to calculate the occupancy rate at each patient's registra- 
tion time. Finally, all variables were aggregated into the research 
database. 

Statistical analysis 

The obtained data was descriptively analysed using numbers 
and percentages for categorical variables and the median and 
interquartile range (IQR) for continuous variables. Differences in 
waiting time and the number of patients simultaneously present 
for patients with safe and unsafe waiting times were analysed 
using the Mann-Whitney U test. The ability of the occupancy rate 
(number of patients simultaneously present at the ED) to discrim- 
inate unsafe waiting times was analysed using a receiver operating 
characteristic (ROC) curve. The ROC curve plots sensitivity against 
(1 - specificity) for all possible thresholds in a binary classification 
task. The area under a ROC curve (AUC) represents the overall dis- 
criminatory ability of a test, where a value of 1 .0 denotes perfect 
ability and a value of 0.5 denotes no ability. 
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When a diagnostic test is based on a continuous variable, the 
ROC curve describes the performance of a model across the entire 
range of classification thresholds. A range of different decision 
thresholds or cut-off values may be investigated in order to iden- 
tify the most preferable one. Sensitivity and specificity have been 
calculated for each of the cut-off values. It is desirable to choose 
a cut-off value that has high values for both sensitivity and speci- 
ficity. In practice, the sensitivity and specificity may not be re- 
garded as equally important. For example, a false-negative 
finding may be more critical than a false-positive one, in which 
case a cut-off with a relatively high specificity would be preferable. 
However, if there is no judgement on the preference between the 
two, then Youden's index (J) may be used to choose an appropriate 
cut-off: J = sensitivity + specificity - 1. In a perfect test, the value J 
achieves a maximum of 1 ; the value J equals 0 when the test has no 
diagnostic value. 

All P values were two-sided, and statistical significance was set 
at a P value of less than 0.05. All analyses were performed with the 
R statistical software package (version 2.15.1). 

Ethical considerations 

Given the retrospective design, this study was not subjected to 
an ethical advisory board. The data used in this study was obtained 
from a database used for business information appliances. This 
database is in compliance with the national law on the protection 
of privacy. 

Results 

Of the 54,280 patients who visited the ED during the study per- 
iod, 44,389 patients were older than 1 6 years of age. A total of 9458 
patients (21.3%) were triaged as ESI level-1 or ESI level-2 and 
therefore included in the study. Of these, 344 patients (3.6%) were 
assigned as ESI level-1, and 9114 patients (96.4%) as ESI level-2. 
Complete data was available in 99.1% of study patients 
(n = 9369). Patients with incomplete data were excluded for fur- 
ther analysis. 

The overall median waiting time for ESI level-1 and level-2 
patients was 5 min (IOR = 4-8) and 12min (IOR = 6-24), respec- 
tively. Unsafe waiting time occurred in 1 6.0% of ESI level-1 patients 
with a median waiting time of 17 min (1QR= 13-23), and in 18.9% 
of ESI level-2 patients with a median waiting time of 48 min 
(IQR = 37-68). A summary of these waiting times is presented in 
Table 1. 

Number of patients simultaneously present at the ED as an indicator 
for unsafe waiting times 

The number of patients simultaneously present differed 
significantly between ESI level-1 patients with a safe waiting time 
(median = 54, IQR = 47-66) and those with a unsafe waiting 
time (median = 64, IQR = 50-75), P= 0.003. Likewise, for ESI 
level-2 patients the number of patients simultaneously present 
differed significantly between patients with safe waiting times 
(median = 55, IQR = 45-65) and those with unsafe waiting times 
(median = 67, IQR = 59-75), P < 0.001. 



ROC analysis for the number of patients simultaneously present 
in detection of unsafe waiting times for ESI level-1 patients pro- 
vided an AUC of 0.625 (95% confidence interval (CI) 0.537-0.713; 
P = 0.003) (Fig. 1). The AUC for detecting unsafe waiting times for 
ESI level-2 patients was 0.740 (95%CI 0.727-0.753; P= 0.000) 
(Fig. 2). 

Threshold for initiation of hospital-wide measures 

Based on the Youden's index, the best cut-off value for detecting 
unsafe waiting times was equal to or higher than 60 patients 
simultaneously present at the ED. With regard to the detection of 
unsafe waiting times, this threshold resulted in a sensitivity of 
0.618 and a specificity of 0.657 for ESI level-1 patients, while for 
ESI level-2 patients the sensitivity and specificity were respectively 
0.738 and 0.627. In order to achieve 90% sensitivity in detecting 
unsafe waiting times, the threshold for the number of patients 
simultaneously present at the ED was 36 for ESI level-1 patients 
(0.093 specificity), and 51 for ESI level-2 patients (0.418 
specificity). 

Discussion 

This observational study, performed in one tertiary referral cen- 
tre, used the occurrence of unsafe waiting times for ESI level-1 and 
level-2 patients as a proxy for the adverse consequences associated 
with ED crowding. An occupancy measure, based on the number of 
patients simultaneously present at the ED, was used to determine 
at what point unsafe waiting times occurred. This indicator was 
found to be a weak predictor for unsafe waiting times in ESI le- 
vel-1 patients (AUC = 0.625; 95%CI 0.537-0.713), but a fair indica- 
tor for unsafe waiting times in ESI level-2 patients (AUC = 0.740; 
95%CI 0.727-0.753). 

Sixteen per cent of ESI level-1 patients had a waiting time of 
more than 10 min (median 17 min), although one can argue 
whether these waiting times are a problem in the first place. Clin- 
ical staff, present during taskforce meetings, confirmed that they 
were not worried about the waiting times for ESI level-1 patients, 
for which two reasons were mentioned. First, these patients are 
generally placed in a resuscitation room prior to registration in 
the ED information system. Second, in cases where ESI level-1 pa- 
tients had to wait for an available resuscitation room, they are 
never left unattended. In other words, the measured waiting times 
for ESI level-1 patients does not always clearly represent the real 
life situation. Given the high urgency (often resuscitation) of ESI le- 
vel-1 patients, it seemed logical that staffs priority was to stabilise 
the patient to the expense of obtaining administrative data. In con- 
trast, a total of 18.9% of ESI level-2 patients had a waiting time of 
more than 30 min (median 48 min). Given the poor clinical condi- 
tion of these patients, the occurrence of such unsafe waiting times 
should be prevented. Best cut-off value for detecting unsafe wait- 
ing times based on the Youden's index was set at 60 patients 
simultaneously present at the ED. In order to achieve 90% sensitiv- 
ity, the threshold must to be set at 51 patients simultaneously 
present at the ED (0.418 specificity). 

The emergence of unsafe waiting times stems from several fac- 
tors within three interdependent components: input, throughput, 
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ESI level-1 5 
ESI level-2 12 


4-8 84.0 
6-24 81.1 
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9 5-16 18.9 48 37-68 
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ESI = Emergency Severity Index. 
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ROC Curve 



Emergency Severity Index: 1 




1 - Specificity 

Diagonal segments are produced by ties. 

Fig. 1. ROC curve showing the predictive power for the number of patients 
simultaneously present in the ED on the occurrence of unsafe waiting times in ESI- 
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Diagonal segments are produced by ties. 



and output (Asplin et al., 2003; Hoot and Aronsky, 2008). Since the 
recognition of the ED crowding problem, most attention has been 
drawn to the number of patients present at the ED. Following As- 
plin's model of ED crowding, this number results from a combina- 
tion of input and output factors (Asplin et al., 2003). As this study 
shows, the number of patients simultaneously present is itself a 



moderate indicator of unsafe waiting times for high urgency pa- 
tients. It can, therefore, only be held partially responsible for the 
occurrence of unsafe waiting times. This suggests that throughput 
factors play an important part in the manifestation of unsafe wait- 
ing times. Throughput is affected by patient characteristics (e.g. the 
level of care a patient requires), organisational factors (e.g. avail- 
ability of treatment spaces, staffing levels, organisation of care) 
and the resilience or the positive ability of ED staff to adapt to 
the consequences of the excessive number of patients present at 
the ED. In order to improve safe waiting times future initiatives 
need to consider these additional causal factors. In other contexts 
the knowledge and experience of nurses and physicians alike have 
been associated with improved outcome. It is therefore not incon- 
ceivable that these factors also apply to crowding. Competent med- 
ical and nursing staff, communication, teamwork, clinical 
leadership and cultural change are some examples which can im- 
prove the resilience of EDs and hospitals to deal with the causative 
factors of ED crowding. 

The second objective of this study was to set a threshold for the 
number of patients simultaneously present at the ED to indicate 
unsafe conditions. This study suggests that the threshold value 
should be set at 51 patients. This value is close to the 59 beds in 
the studied ED (cubicles, observation ward and overflow locations). 
This might suggest that the occupancy level, as suggested by Hoot 
and colleagues, can be a meaningful measure for real time crowd- 
ing prediction (Hoot et al., 2007). The study by Hoot and colleagues 
investigated the potential for monitoring current and near-future 
ED crowding by using 4 measures: the Emergency Department 
Work Index (EDWIN), the National Emergency Department Over- 
crowding Scale (NEDOCS), the Demand Value of the Real-time 
Emergency Analysis of Demand Indicators (READ1), and the Work 
Score. The ED occupancy level was used as a control measure for 
baseline comparison. The occupancy level was calculated using 
the following formula: 100*P bed /B t . Where P bed = number of pa- 
tients in licensed beds and overflow locations, such as hallway 
beds or chairs; and B t = number of licensed treatment beds. They 
found the occupancy level having highest discriminatory power. 
None of the measures provided substantial advance warning be- 
fore crowding at low rates of false alarms. 

Limitations 

The findings of this study are subject to several limitations. 
First, the study was performed in only one Belgian tertiary referral 
university hospital. As a result, findings cannot be generalized to 
other EDs or countries. However, the suggested method could be 
of interest for many EDs. Second, given the retrospective design 
of the study, data were based on historical figures. The variables 
used in our analysis are prone to changes due to alternations in 
workflow, architectural design, etc. Therefore, it is important to 
evaluate the threshold values at regular times, and especially after 
the implementation of interventions aimed to reduce waiting 
times. Third, given the retrospective design it was difficult to pin- 
point the exact time a patient had to wait before being evaluated 
and treated. The time to placement in a treatment cubicle was seen 
as the most accurate and available indicator of the actual waiting 
time. We do recognise however that some patients already re- 
ceived evaluation and even treatment before placement in a treat- 
ment room. Last, the assessment of patient safety is a huge 
challenge. Measuring patient harm resulting from healthcare man- 
agement is much more difficult as compared to the evaluation of 
clinical quality outcomes (Vincent et al., 2013). First, not all harm 
is immediately visible in the ED as patients could be transferred 
to a hospital ward or discharged to their homes. Second, some ad- 
verse events could be resolved within the ED and could therefore 
not be detected using retrospective data. Third, there are relatively 
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few ways in which things can go right but innumerable ways in 
which things can go wrong. In other words, patients can be harmed 
in many different ways. As such, this study attempted to measure a 
universe of possibilities that could only be partially specified in ad- 
vance. Therefore, we did not predefine unsafe waiting on the basis 
of damage incurred. Instead, we pragmatically defined unsafe 
waiting times as the point at which quality of care would poten- 
tially decrease and adverse events due to treatment delay could oc- 
cur. This interpretation is in line with current understandings of 
unsafe conditions (Aven, 2013; Hollnagel, 2013). 

Conclusion 

In conclusion, this study shows that the number of patients 
simultaneously present at the ED is a moderate indicator of unsafe 
waiting times for ESI level-2 patients. In our ED, we set the thresh- 
old to 51 patients simultaneously present to predict unsafe waiting 
times with a sensitivity of 90%. The results of this study suggest 
that in order to improve safe waiting times the focus towards the 
number of patients present at the ED should be expanded to in- 
clude other causal factors of unsafe waiting times. Improving 
organisational resilience by strengthening the ability to adapt to 
the consequences of the excessive number of patients present at 
the ED seems a meaningful proposition. 
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